In 2012, the number of connected mobile devices in use, including mobile phones and tablets, exceed the population of our planet. Market forecasts indicate that by 2016 there will be over ten billion of Internet-connected mobile devices in the hands of end users, of which approximately eight billion will be smartphones and tablets. Accordingly, digital lifestyles of billions of people will become increasingly dependent on their use of smartphone and tablet applications.
One of the largest smartphone application categories is related to use of phone cameras. According to industry statistics, over 83% of the 1.78 billion mobile phones shipped in 2012 and nearly all smartphones have a camera. Current smartphones have relatively good quality cameras, stimulating quick growth of scanning applications. Scanning applications on smartphones are already used by hundreds of millions people worldwide. Two categories of smartphone scanning activities include taking, storing and processing photographs of paper documents and scanning printed or otherwise displayed matrix (two-dimensional) barcodes for mobile tagging. According to recent market surveys, a smartphone or tablet camera for document capture, and cloud file services, are used by more smartphone and tablet users than mobile scanners, and mobile printing services. Additionally, a December, 2011 survey indicated that approximately 33% of smartphone owners in Japan, 20% of US smartphone users and 14% of smartphone users in EUS, the five most populated European countries, have scanned QR codes, which is only one (albeit the most popular flavor) of the over 70 types of currently existing matrix barcodes.
Online services and multi-platform software such as the Evernote service and platform by Evernote Corporation of Redwood City, Calif., ABBYY by ABBYY Group of Companies, the Dropbox service by Dropbox, Inc. and many other solutions offer image storing, advanced processing and search within images of photographed documents. In particular, the Evernote Service offers indexing and search of handwritten documents, which makes smartphone document scanning an attractive and potentially ubiquitous method of the paperless life. As to barcode scanning, recent surveys revealed that over 75% of US retailers are offering matrix barcodes to their customers. In addition to the three most popular applications of mobile tagging, namely, obtaining product information (most popular in US and Europe), receiving discount offers for goods of services (a dominant QR code application in Japan) and getting event details, there are numerous casual, educational and other uses of matrix barcodes, such as museum guides, searching for lost things, gaming, dating and many other uses.
Notwithstanding significant advances in smartphone camera quality and processing power of mobile devices, capabilities and scope of online services, and their extended document and image processing features, smartphone scanning still faces significant challenges for both categories of document and barcode scanning. In some cases, obtaining quality images of handwritten pages from a camera phone and other photos is a difficult task. Photographs of handwritten pages are subject to variable lighting conditions, perspective distortion, background effects, bending toward notebook edges, etc. Techniques for correcting real-life photo images with shadows, reflections and distortions; identifying page boundaries; separating handwriting from paper background; correcting perspective distortions and curved edges, and other similar tasks have been explored by many vendors with varying degrees of success. In particular, the tasks of unique identification of a page of a notebook, and reconstructing page boundaries have been traditionally solved by adding barcodes or cropping marks, such as a page frame, corner markers and other distinguishing and location-bound page elements. Such approaches make page identification results vulnerable to occasional reflections and shadows, interference with unrelated objects that may shield portions of photographed pages, etc. In some cases, note-takers' instructions for note filing and note related actions at the time of writing are different for different pages of handwritten documents. Examples may include tagging, merging handwritten pages, communicating portions of note content to different people, etc. These actions may be reoccurring from note to note and may be easily forgotten if not memorized instantly. However, existing paper based note-taking systems lack simple, easily recognizable and convenient means of indicating actions that can be instantly converted into the digital form identifiable on smartphone photographs of notebook pages.
Analogously, the usage of QR codes and other rasterized identifiers is not without problems. One disadvantage of such marks lays in their conflict with the aesthetics of product design. For example, an image of a QR code with a mediocre information capacity of 98 characters has a minimal linear size of 48 mm (1.9″) for reliable scanning from a comfortable distance of 300 mm (12″) that would secure a subsequent accurate decoding by the smartphone software. For many small-size goods carrying aesthetical functions, the presence of a relatively large black-and-white square patch might be a considerable design problem. For example, an online marketing guide to QR codes repeatedly warns against an inappropriate use of QR codes on promotional items: “Putting the code on the front makes the t-shirt unattractive . . . . ”
Another potential issue with processing matrix barcodes is that, in spite of generally reliable error correction codes incorporated into such identifiers, their recognition completely depends on the location of characteristic elements within the codes, such as the bullseye cropping marks at the angles of a QR code or the black L-shaped finder pattern border and the alternate timing pattern border in a Data Matrix barcode. In a non-commercial photographing environment, these elements may be easily obstructed and may be quite sensitive to lighting conditions, which creates an increased risk of losing the recognizable codes which are present on the photo but remain unidentified by the software. It also takes time for a user to locate and target matrix barcodes with a phone camera, which doesn't align well with other photographing activities where the aesthetical aspect is important and where image processing tasks are involved.
Accordingly, it is desirable to design streamlined methods for automatic identification of photographed objects, including correction methods for images of paper pages with handwriting, and for performing image and data processing tasks that combine reliable recognition and identification with aesthetical attractiveness of photographed scenes.